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In this paper, we present sequential codes which have interesting 
properties in three respects. First, these codes may be used to achieve 
tow redundancy (e.g., bandwidth compression through coding) by 
employing a multiplicity of variable-length codes to encode transitions 
between successive source symbols. Second, the coding complexity is 
surprisingly tou\ Third, many of these codes have exceedingly good 
intrinsic recoverability properties following errors. These codes com- 
pare favorably with a difference code environment m which the dif- 
ferences between successive source symbols are encoded. The scope 
of the sequential codes presented here includes* but is much wider than, 
difference code schemes. Where comparable, the sequential codes have 
slightly greater complexity and may have lower redundancy. They 
normally have vastly superior error recovery. These codes are ap- 
plicable in situations such as video transmission in which the message 
source is highly correlated and where errors can be tolerated for a short 
period of tune. 

I. INTKOOICTION 

In several previous papers, Ihc author has pursued (wo apparently 
separate paths of development. The first patli involves classes of 
slightly suboptimal variable-length prefix codes 1 - whose self-synchro- 
nizing abilities are vastly superior to the optimal (Huffman) codes 
which minimize redundancy. The second path involves self-synchroniz- 
ing sequential codes using in format ion- lossless sequential machines as 
encoders and decoders, 1 * In this paper, these two paths of development 
are joined- The result produces highly efficient sequential codes (with 
low redundancy) which have good self*synchronia;ing abilities and 
surprisingly low decoding complexity. These codes are applicable in 
situations in which the message source is highly correlated. 

051 
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1. 1 Difference Codes 

Given an environment in which successive source signals are likely 
to be similar (or at least strongly correlated), considerable compres- 
sion can be obtained by encoding the level difference between succes- 
sive quantized signal levels (temporally or spatially). Such is the 
situation in a video picture environment, for example, with respect 
to adjacent points horizontally or vertically, or even to the same point 
in successive frames. By encoding differences, however, any error in 
quantizing, encoding, transmitting, decoding or reconstructing the 
image tends to persist. That is, once an error has occurred in a signal 
level, subsequent levels will continue to be in error by the same offset, 
unless terminated by boundary effects or by compensating errors. In 
an encoding of level differences between successive points in a line, 
for example, errors tend to propagate until the end of the line; in an 
encoding of level differences between the same point in successive 
frames, on the other hand, errors may continue forever. In order to 
prevent such error effects from propagating indefinitely, it may be 
necessary to terminate the propagation forcibly, for example by 
transmitting periodically the set of signal levels for the entire frame 
("replenishment") rather than tJieir frame- to- frame differences. Thus 
the use of difference coding for compression may be compromised by 
the need to rcsynchronixc. This is true in general of frame- to- frame 
difference codes. An example of the use of such codes in a differential 
pulse-code modulation environment is given in Ref. 5* 

II. SELF-SYNCHRONIZATION 

The main purpose of this paper is to present codes which have 
compression capabilities at least as good as difference codes, along 
with roughly comparable decoding complexity, as well as having 
rather remarkable intrinsic self-synchronization properties. (These 
codes are in fact much more general than difference codes in terms 
of compression capabilities.) These codes recover quickly from the 
effects of errors in that arbitrary errors give incorrect results for a 
period of time, after which the entire system resumes correct operation 
without any explicit effort, (Note that self-synchronization is a prop- 
erty of the coding scheme, and should not be confused with video 
picture frame synchronization.) It is important to note that in such a 
scheme the errors are not corrected (in the sense of error-correcting 
codes) ; instead errors are tolerated, with the expectation that their 
effect will cease quickly. Video coding is an example where such an 



SELF-SYNCHR0XIZ1MS CODING 953 

approach is reasonable since loss of information for a short period of 
time can often be tolerated. 

2.1 Selj-SijnehronizQtion in Variable-Isength Codes 

The use of variable-length codes for redueing redundancy is well 
understood. For example. D. A. Huffman* shows how to obtain a code 
whieh minimizes! (he transmitted information for a given independent 
distribution of source symbols. However, there has been relatively 
little quantitative concern for the effects of errors on these codes. In 
earlier papers 1 -* the author has shown that a slight sacrifice in efficiency 
(i.e., a slight increase in transmitted information! can be rewarded 
with tremendous gains in self-synchronizing capability. Some of this 
work is required here and is reviewed briefly, although from a differ- 
ent viewpoint. 

A code is a collection of sequences of digit* (code digits) , each 
sequence being called a code word. Code text is obtained by concat- 
enating code words. An encoding is a mapping of source symbols S(i) 
onto code words IF(fl. A code is a prefix code if and only if no code 
word occurs as the beginning (prefix) of any other code word. Thus 
in prefix code text, a code word can be decoded as soon as it is received, 
even though there are no explicit interword markers. A code is 
exhaustive if and only if every sequonen of rode digits is the prefix of 
some code text (i.e., of some sequence of code words). (A uniquely 
deeodable code must be a prefix code if it is exhaustive/) A sequence 
of code digits is a synchronizing sequence for a given code if the 
occurrence of the end of that sequence in (correct) code text must 
corresj>ond to the end of a code word (although not necessarily to a 
particular code word), irrespective of what preceded that sequence. 
M. P. SchUtzcnbcrger" and E. X. fiilhert and E. F. Moore 7 have shown 
that most exhaustive prefix codes tend to resynchronize themselves 
following loss of synchronisation (e.g„ after arbitrary errors, or at 
start-up). If an exhaustive code has at least one synchronizing se- 
quence, then the code tends to resynchronize itself following errors 
with a finite average delay (assuming a suitable randomness). All 
codes considered here are exhaustive unless explicitly stated otherwise. 
Note that resynchronization is an intrinsic property of the code, and 
no externally implied synchronization is required. Synchronization 
following ambiguity occurs as a result of any synchronizing sequence 
occurring naturally in code text. 

As an example, consider the code of Fig, 1. consisting of the five 
code words 00, 01, 10, 110, 111. The tree of Fig. 1 may be interpreted 
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Fig. 1 — A simple prefix codr, 

as the state diagram for a sequential machine 11 which detects the end of 
a code word whenever the initial state 1 recurs. In this figure (and 
throughout this paper) a M 0" code digit corresponds to left-downward 
motion, a "1" to right-downward motion, These digits arc the inputs 
to the sequential machine. In Fig. 1, state 4 is equivalent to state 2 
(in the usual sequential machine sense ), since the respective next 
states are equivalent, for each input digit* Thus Fig. 1 represents a 
3-state machine whose recurrence of state 1 indicates the end of a code 
won! in text. 

The synchronizing diagram 1 ,0 for the code of Fig. 1 is shown in 
Fig. 2. It is obtained from Fig. 1 by examining the set of next states 
resulting from each input digit, beginning with the set of all states, i.e., 
total ambiguity. For example, a "0" digit can lead only to state 1 or 
2, and a "1" can lead only to state 1, 2 (formerly 4|, or 3. Given the 
set of states 1, 2, a "1" can lead only to state 1 or 3. In this way it is 
seen that the sequence 0110 always culminates in the occurrence of 
state 1, irrespective of the actual state at the beginning. (This is indi- 
cated in Fig 2 by the dark path.) Thus 0110 is a synchronising se- 
quence for the code. (Note that its occurrence in code text following 
ambiguity does not imply the conclusion of a particular code word; 
either 10 or 110 could be involved.! There is an infinite set of syn- 
chronizing sequences described by Fig. 2, including as other examples 
01 1U 10 and 10110- 

If a sequential machine (or a code) has at least one synchronizing 
sequence, then it tends to resynchronize itself with probability one, 11 
assuming all input sequences are possible. In order to obtain a measure 
of how well text for a given code is self-synchronizing following arbi- 
trary errors, the code digits "0 M and "1" are assumed lo occur inde- 
pendently and equiprobably. This is in fact a meaningful and useful 
assumption under various reaMife circumstances, even when it is only 
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Fig. 2— Synchronizing digram for the code of Fig- 1, showing unresolved 
ambiguity, 

approximately valid. (This assumption holds exactly whenever each 
code word occurs independently with its characteristic probability 2"*, 
where d is the length of the code word in digits.) Assuming this ran- 
domness of and 1 in code text, the synchronizing diagram of Fig. 2 is 
redrawn in Fig. 3 to show that on the average 7=16 digits of code text 
are required for code text to resynehronize itself following arbitrary 
errors. (This computation may be done in several ways 1 which are not 
relevant to the present discussion. Note that the randomness assump- 
tion implies that at each node in the synchronizing diagram the resi- 
dual lag is one more than the average between the residual lags of the 
two nodes below.) In general, the synchronization tag (or, simply, the 
lag) / of a (prefix) code is defined as the average number of code 
digits until synchronization can be guaranteed to the end of some (not 
necessarily known) code word following total ambiguity, assuming 
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Fig. 3 — Copy of Fig. 2, showing tag at each node. 
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randomness of "0" and "1" as above. Thus the lag is the average length 
of the synchronizing sequences. (It is also convenient to speak of the 
actual lag, the same average but based on the probabilities of the actual 
source distribution rather than on random text. If the randomness 
assumption is not roughly applicable in a given case, then it is neces- 
sary to investigate the actual lag. If the likelihood of the various 
synchronizing sequence* actually occurring in text is smaller than under 
the randomness assumption [e.g., because of constraints on the source 
symbols], the actual lag is larger than /. However, in such cases it is 
often possible to change the encoding slightly to assure that the actual 
lag is less than I, without adversely affecting compression. It may 
also be possible to use a different code with the same set of code-word 
lengths whose lag is less. For example, the code 00, 01, II, 100, 101 
has / = 5, a marked improvement over the code of Fig. 1, with / = 16; 
for purposes of compression, these two codes are equivalent. In general, 
the randomness assumption and the resulting lag are quite useful.) 

[Synchronizing sequences also exist for uniquely decodable non-pre- 
fix codes. For example, consider the code 00, 01, 11, 001, 011." The se- 
quence 10 in code text guarantees that a code-word end occurred 
between the "1" and the "0"; the lag is 4. Since such codes may require 
decoding delays (in some cases infinite) beyond the end of their code 
words, they are of little practical significance. (This paragraph and 
others delimited by square brackets may l>e omitted on casual read- 
ing.) ] 

Codes vary widely in their ability to resynchroniae code text. For 
each number n of code words, there is a code with / = 2 (the "best"), 
and a related code with / = 2'*- 1 * - 2 (the "bad" code). These codes 
are shown in Fig. 4 for each n ^ 9. along with a few other examples. 
(Note that the "best" code and the "bad" code for each n have the 
same set of code-word lengths, and are thus equivalent with respect to 
compression considerations.) The only finite exhaustive codes with / = 
*> known to Schutzenbcrger* (and to the author) arc the block codes 
(e.g., the fixed-length codes (a), (b) in Fig. 4) and three classes of 
non-block codes: the codes with greatest common divisor of their code- 
word lengths greater than one (e.g., code (c) in Fig. 4), the uniformly 
composed codes C f obtained by concatenating / times the code words 
of some code in all combinations (e.g., code (d) in Fig. 4, composed 
from 0, 10, 11 with / = 2), and Schtttzenberger's "anagrammatic" 
codes" which when scanned backwards are also prefix codes (e.g., code 
(c) in Fig. 4). Note that block codes have the properties of all of these 
three classes. (By sacrificing exhaustivity [and optimality], i.e., by 
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Fig. A — Prefix codes with external lugs for n % ft. 

eliminating at least one code wonl from a code, the lag is never in- 
creased; in some cases / is reduced substantially, |A slight extension of 
the definition of the lap is necessary for non-exhaustive prefix codes to 
handle sequences which cannot arise. | Thus there are synchronization 
advantages of noncxhaustivc codes.! 

The only finite exhaustive codes .known to the author which have 
2<"-n — 2 < / < « for any n are the two codes (0 and (gl in Fig. 4. 
Excluding these two codes and the ''had" codes, all remaining finite 
lag codes seem to have / < 2'"" 1 * - 2, for all n; the worst of the re- 
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raaining codes seem to be far from the "bad" code bound (/ = 5.5 for 
n = 5, / = 12 for n = 6). Nevertheless, as n increases, the lag of an 
arbitrary code may be quite bad. Since prefix codes offer compressions 
which become more spectacular as n increases, this would be distressing 
were it not for the smalt-lag infinite systematic codes of Refs. 1 and 2. 
Since these are helpful in the construction of sequential prefix codes, 
they are summarized next. 

2*2 Infinite Cadet with Good Synchronization Properties 

The "systematic" prefix codes of Ref, 1 are infinite codes generated 
by sequential machines whose synchronising projiertics guarantee that 
the codes will themselves have good synchronizing properties* (These 
codes have properties suggested by B. Mandelbrot.") When truncated, 
these codes give highly efficient (but not optimal because they are no 
longer exhaustive) encodings with self-synchronizability far exceeding 
that of the optimal (Huffman*) encodings, A typical reduction for a 
code for English words with n = 5537 is from I 3> 1,000,000 for an 
optima! code down to / = 10.7 for a systematic code which is within 
3 percent of the optimal code in terms of compression. (The resulting 
compression is about a factor of three better than a block coding of 
the English letters, due to the redundancy of the language.) High- 
efficiency compression using these codes is discussed in Rcf. 1. 

Several examples are given in Fig. 5. The convention of this and 
succeeding figures is that a terminal node without fln arrow indicates 
the end of a code word. A terminal node with an arrow indicates the 
occurrence of a state shown elsewhere in the diagram. If there is no 
label on the arrow, the corresponding state is that occurring at the top 
(the root) of the diagram. The first code la) is a "definite" code, 2 with 
any occurrence of the sequence "10" in code text indicating the end 






tbl 

Fig, 5— Examples of automatic prefix code* with small la^a /. (a) Definite 

code, / = 4 r L = 4; N(d) =01234587 (b) Trec-baeed code, 7 = 8. 

L = 6: N{d) := 1 1 2 3 5 8 13 .... (c> General systematic. / = 6, L = 4: 
N{d) =01234867 
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of a code word. A definite rode is one in which a code-word end occurs 
in code text if and only if one of a finite set of finite sequences occurs; 
in the example, the sequence "10" forms the set The other codes are 
not definite, but nevertheless have small lags. The number N{d) of 
code words of each length d is given for each code. The function A r (d) 
is usually called the structure function of the code. The average code- 
word length (assuming randomness of *'0" and "l w in code text) is L t 
the sum of d N{d) 2~* over all d. (If each code word occurs inde- 
pendently with its characteristic probability, then L is equal to the 
entropy // of the source distribution.) Note that / S L for all prefix 
codes, with equality if and only if the code is definite. That is, on the 
average, one code word is sufficient to resynchronize a definite code, 
longer Wing required for a non-definite code. 

It appears that asymptotically almost all [completely specified, 
strongly connected, deterministic) sequential machines have synchro- 
nizing sequences; for those that do, the resulting systematic prefix 
codes are self-synchronizing (i.e., / < oo). The spectrum of values of 7, 
however, is wide. As is the case with exhaustive finite prefix codes, the 
infinite codes with / = oo may be of several "classes. There exist codes 
with a non-unit greatest common divisor of their code-word lengths 
(e.g., Fig. 6a), with uniform composition (e.g., Fig. 6b), and with the 
anagrammatic property (e.g., Fig. 6c). Unlike the case for finite ex- 
haustive codes, infinite exhaustive codes can easily be constructed 
which belong to none of these three classes (e.g., Fig. 6d). (Uniformly 
composed codes are studied in Ref. 13.) 

[The worst value of / for n code derived from a synchronizable 
r-state machine appears to be about / ^ (r — £) (2 r — 2) + 1 ; that is, 
just about a factor of r worse than the "bad" code of Fig. 4 with r 
states, r = n — I. A. E. I-nemmel and B. Rudner' 4 exhibit for each r a 
machine whose shortest synchronizing sequence is (r — 1)*. For all r- 
state machines with synchronizing sequences, the best bound known 
to the author for the longest synchronizing sequence is that given by 
M. A. Fischler and M. Tannenbaum 11 : (r — 1)*, exact for r ^ 4; 
<r* ■- r)/6 for small r & 5; 1 lr»/48 for large r. On the other hand, 
there are many machines with extremely short synchronizing se- 
quences and small values of /.) 

111. SYNCHRONIZATION IN SEQUENTIAL PREFIX CODES 

For purposes of this paper, sequential coiling implies the use of a 
sequential machine for the encoder, and a corresponding sequential 
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(*> (b) (Cl (d) 

Fig. 6— Examples of systematic codes with infinite lag (/ = <o)* (a) Greatest 

common divi*or2; N(d) = 01030-90 27 81 (b) Uniform composition; 

/ = 2. N(d) =01234567 (c) Anwammatic; N(d) =0222222 

(d) None of the three clams; N(d) = 1 1 2 3 5 S 13 21 

machine (the inverse or "quasi-inverse") for the decoder. 1 * In an 
earlier paper, 3 the author explored the use of particular input sequences 
to the encoder (synchronizing input sequences) each of which synchro- 
nizes the encoder to a particular known state. Also involved are partic- 
ular output sequence* from the encoder (synchronizing output 
sequences) whose presence in code text (in the absence of errors) guar- 
antees the occurrence of a particular state of the encoder at a particular 
point in the code text. Synchronizing output sequences correspond to 
the synchronizing sequences for prefix codes. 

Given an encoder with both synchronizing input sequences and 
synchronizing output sequences, the entire system is self-synchronising 
on the average. That is, following some arbitrary errors, two things 
happen. First the encoder resynchronizes itself and begins to encode 
correct text again. Then the decoder tends to resynchronize itself with 
the encoder (the synchronizing output sequences for the encoder act 
as synchronizing input sequences for the decoder), and correct decod- 
ing resumes. This occurs spontaneously as an intrinsic property of the 
coding system, with no externally imposed resynch ionization required. 

A (first-order) sequential encoding is a mapping of symbols S(i) 
onto code words w(i | j) where the code word selected depends on the 
previously encoded symbol S(j) as well as on S(i). If the set of code 
words {w(i | ;) ) for each j is a prefix code, then the set of code words 
{w(i | ;) ( for all i, j is a sequential prefix code. For such codes a syn- 
chronising sequence is a sequence of code digits the end of which must 
correspond to the end of a code word (possibly unknown) resulting 
from a known symbol S{i) t irrespective of what preceded that se- 
quence. Thereafter subsequent decoding is correct, irrespective of the 
initial ambiguity. The remainder of this paper is concerned with 
sequential prefix codes, and investigates their compression, decoding 
and self-synchronizing properties. (The development is also applicable 
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to higher-order sequential encodings, with the code word depending on 
the present message Sti) and some finite number of previous messages.) 

As an example, consider the sequential encoding given by Table L 
A, B, C and D represent four source symbols S{i), i = 0, 1, 2, 3, where 
i is the level of the symbol. This is an example of an encoding in which 
the code word w(i*j) to be transmitted is a function of the cyclic 
difference between the level of the symbol S(i) to be encoded (column 
headings) and the level of the symbol SI;) just previously encoded 
(row headings): w{i\j) = W(k) t where k — i - } (mod 4(. This 
encoding is thus a difference encoding. Note that, irrespective of the 
choice of the code (ir(fc)}, there is always ambiguity in decoding as 
soon as an error is made. If for example S(2) is decoded instead of 
5(1) as a result of a transmission error, subsequent decoding will 
consistently produce S(i + 1) instead of S{\) where i + 1 is modulo 
4, as long as further errors do not compensate for the original errors. 
(Throughout the paper, all additive operations involving i and ; are 
modulo n>) 

As a second example, consider the sequential prefix code of Table II. 
In this example four different prefix encodings w(t|;) are used for 
; = 0, 1, 2, 3, depending upon the symbol S(j) previously encoded. 
For example, if "A" was just encoded, then A, B, C, D arc encoded 
as 0, 11, 100 ( 101, respectively. Thus any symbol S(i) as input to the 
encoder acts as a synchronizing input sequence. (The encoder is a 1- 
definite machine, 1 with its output being a function of the present input 
symbol and the previous input symbol.) 

A state diagram for the decoder is given in Fig. 7. The states A, B, 
C, D represent the successful decoding of these four symbols, while the 
atates a, b, c, d, a' f b'. c 7 , d' are intermediate states- The state diagram 
is shown in four pieces, which fit together as the upper-case letters 
indicate. The synchronization diagram for this state diagram is shown 
in Fig* 8, Us construction follows the usual technique 3 * 10 and is similar 
to the synchronization diagram for prefix codes (cf. Fig. 3). The top 
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Table II— A Good Sequential Prefix Code 
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node corresponds to the set of all states. Each terminal node corre- 
sponds to the occurrence of the end of a code word resulting from a 
particular symbol S{i). Given a "0" input, the next state must be one 
of A, C, b, d, a', c', for example, beginning with the set of all states. 
The synchronising diagram of Fig* 8 results after noting several 
equivalences {e.g., AbdaV with ACbduV), 

From the synchronizing diagram of Fig. 8 it is seen that the se- 
quence 0011 can arise in code text only if the corresponding source 
sequence ends in a "B'\ Thus 0011 synchronizes the decoder to the end 
of a code word corresponding to the symbol "B" irrespective of what 
preceded it; similarly 00101 synchronizes to "D", 1100 to "C", and 
11010 to "A". Assuming and 1 arc random in the above sense, it is 
easily shown that synchronization results from total ambiguity after 
an average of J = 7,67 digits. The sequential synchronization lag J 
is the average number of code digits until the end of a code word is 
achieved corresponding to a known symbol; that is, J is the average 
length of the synchronizing sequences. In this example, the occurrences 
in code text of the encoded versions of CB, CD, BC and BA imply 
synchronizing sequences for the decoder. (For example, note that CB 
is encoded as 0011 following a "B" or "D'\ as 10011 following an "A", 






A D 




Fig. 7— State diagram for the decoder for the code of Table III. 
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ABCDabcda'bVd' 




Fig, &— Synchronizing diagram for the decoder of Fig. 7, / = 767, J' = 3, 
J" = 4-67, 



and as Oil following a "C"; in the last case, Oil must have been 
preceded by a "0*M 

In the example, any sequence ending in 00 or II (cf. Fig, 8) guar- 
antees the end of a code word, although not the code word for a known 
Symbol. For purposes of this paper, synchronization to the end of some 
(unspecified) code word is called first-stnge synchronisation. Synchro- 
nization to the end of a code word corresponding to a particular symbol 
is called second-stage synchronization. The former is of concern in 
prefix codes, and both are of concern in sequential codes* In rare cases 
(particularly asymmetric ones), the second stage is achieved simul- 
taneously with the first stage. In many useful cases, however, they 
may be treated independently. (In all but one example given in this 
paper, second-stage synchronization implies a particular known code 
word as well as a particular known symbol.) 

(A word of caution is again needed regarding the randomness as- 
sumption. Again, the actual lag may be defined in terms of the actual 
probabilities. If synchronizing sequences occur naturally in code text, 
then the lag J is a fair estimate of the actual lag. If the sequences occur 
only as a result of unlikely sequences of symbols, then the lag J is 
smaller than the actual lag. However, in such cases the encoding can 
often be altered so that J is realistic. In general t it is desirable to have 
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codes with widely distributed likely synchronizing sequences, as in 
Fig. 8, rather than being totally dependent on a few obscure sequences.] 

IV, CODING COMPLEXITY 

It is clear that the choice of the code of Table II (with J = 7.67) 
is (infinitely) superior to the code of Table I (with J = oc) in terms of 
resynchronizability. If W(k) = 0, II, 100, 101 in Table I for k = 
( l r 2, 3, then the codes of Tables I and II are identical with respect 
to compression capabilities. The significant concern remaining is the 
computational complexity of the encoder and the decoder for the code 
of Table II. Intuitively, one might expect that an n-state sequential 
prefix code would be almost n times as complex as a difference code in 
terms of its encoding and decoding circuitry. Somewhat surprisingly, 
codes are developed below for which the complexity is essentially the 
same as the difference codes, while attaining good synchronteability 
and compression. 

Examination of the code xv(i | ;) of Table II shows that the prefix 
code w(\ | 1) for state B {} = 1) is the binary complement of the code 
w(i | 0) for state A (} = 0), cyclically shifted by one word: w{i 1 1) = 
Vf{i —10), Further, the code w(i \ 2) for state C {j = 2) is related 
to w(i | 0), having the same code words but with a different mapping. 
In particular, w(i\ 21 = w{2 — r| 0)- Finally, the code for state D 
(j = 3) is the shift of the complement of w(x J 2), or the complement 
of the shift, and thus w(i\3\ = vf(B - t|0). Thus all four of the 
prefix codes are closely related to any one of them. 

The code w(i \ j) as a function of u?(t | 0) is summarized in Table 
III for each j. The structure of the code is somewhat more transparent 
when related to the difference code of Table I, with W(k) = 0, II, IO0, 
101 for k = 0, 1, 2, 3 and k = i — j (mod 4), In this case, w(i \ j) = 
W{k) f W'{k) t JF(-fc). W(-k) for / = 0, 1, 2, 3, respectively. The 
encoder and decoder for Table II arc thus easily specified in terms of 
the difference code. If (p, q) is the binary representation of j as shown 
in Table III, then W{k) is replaced by W(-k) ifp = l f and the result 
is complemented if q = 1. The encoder for the difference code is shown 
in Fig. 9, while the encoder for the sequential prefix code of Table II 
is given in Fig. 10. ("A" represents a one-digit delay.) It is seen that 
an AND gate (•) and two EXCLUSIVE OR gates (©) represent the 
marginal cost of encoding the latter code, compared to the difference 
code. The same is true of the decoder, which employs precisely the 
same set of gates. 
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Tables III— Summary of Encoder and Decoder for the Code of 

Table II 



i 


P 


9 


Code w(i\j) 


ii 
1 

2 
3 





1 
1 



1 


1 


w<i|0) - v>W) - W(k) 
«Ki]l) - «.'(.-l|0) - ir'(ft) 
w(t|2) - w(2-i|0) = W(-k) 
vffi]3) - w'<3-t|0) - W'(-k) 



V. CONSTRUCTION OF GOOD SEQUENTIAL PREFIX CODES (WITH SMALL LAGS) 

The synchronizing properties of sequential prefix codes fall into two 
classes, those which depend on the individual choice of the encoding 
w(*|j) for each t f j, and those (in varying degrees) which are inde- 
pendent of the actual choice of w(i|j). (The encoding of Table I, 
for example, has J = co irrespective of the w(i\ }).] Such properties 
are said to be choice-dependent and choke-independent for these two 
classes. An example of how choice-independent properties may be 
treated separately is given by the following theorem. 

6.1 Very Bad Codes (J = <*) 

Theorem 1: A sequential prefix code has J = co in each of the 
following cases: (a) if there are precisely n distinct code words among 
the n x n [w(i \ j)} M each of which occurs exactly once for each t 
(a "Latin square" code); (b) if for some value of s (0 < s < n) the 
relation w(i + s\j + $) = w(i | }) holds for aU i and ;. 

Proof: Case (a). Consider an ambiguity between any two symbols 
which could have led to a given code word. Then any subsequent code 
word could have resulted from either of two distinct symbols. Thus 
ambiguity is never reduced, and J = » irrespective of the choice of the 
n independent w{i \j) t (Note that if the prefix code for any j (the same 
code words, but a different encoding for each j| is self-synchronizing 
(/ < *>), then it is possible to reduce ambiguity to the end of some 
(unknown) code word, hut no further. Recall that in the definitions, a 
distinction is made between the code (the set of code words) and 
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Fig- ft— Encoder for the difference rode of Table I. 
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Fig. 10— Encoder for th* aequentinl prefix code* of Tabic II, 

the encoding (tlie mapping between code words and source messages),] 
Case (hi. Without loss of generality, suppose that & is the smallest 
value of s for which w(i + s | ; + 9) = u*(i | ;) for all tand >, (If * is 1, 
the code is a difference code.) Then w(i + gs J j + gs) = w(t | j) for ail 
i, j t for every integer g. Let u be the smallest value of g > for which 
gs « (mod n). Then since « is as small as possible, it follows that su 
= n, i.e., 5 divides m Thus there are * prefix codes w(i i),e*g., for j = 

0, 1 , •••,*— I , which completely determine the prefix codes for j = $, 
a + I, ■••, n — 1. That is, the same prefix code (shifted) occurs for 
each ; = gs + ft (mod n) for g = 0, 1, - * \ (n — *)/«, for any fixed 
value of A, 5 ft ^ s — 1- Thus there must always remain an ambi- 
guity among all the symbols with i = g$ + h (mod n) for g = 0, 

1, -•-, (n- s)/$, for any particular value of A, ^ ftgs- I. There- 
fore J is infinite again, irrespective of the choice of the m independent 
w(i\j). QED. 

[Case fb) of Theorem 1 can easily be extended to include cases for 
which w(i + s\j + t) = w(i\j) for all i, j: if & and ( are each 
relatively prime to fl, or more generally if s and / have the same order 
in the field of integers {Le, f if u = t\ where u and v are the smallest 
non-zero values for which us s (mod n) and vl = (mod n)}. As 
these cases are less natural to the present environment, they are men- 
tioned parenthetically. I 

The difference codes satisfy both cases (a) and (b). Another example 
of case (b> is given in Table IV. If e, /, g, h are replaced by 6, c f d f a, 
respectively; this example also satisfies case (a), and is a "sum" code 
rather than a difference code. 



5.2 Properties 0/ Good Codes 

It is highly desirable that sequential codes {w{i | jl ) have consider- 
able structure, in order to simplify encoding and decoding, to simplify 
the analysis of synchronizing properties, and to facilitate the construc- 
tion of good large codes. Several intuitively evolved properties have 
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b«n examined which are Found to contribute considerably to this 
desired structure. Fir$tly t to simplify encoding and decoding, the num- 
ber m of distinct prefix codes Inot counting complement* ( used to form 
a sequential code should lie small lone or at most two). If a prefix 
code and its complement are both used, the sequential code is comple- 
mented. In the complemented code of Tables II and III, m is one 
since the prefix code for each j is W{k) or its complement. Since the 
circuitry required is up to m times as complex as when m — 1, large 
values of m are to be diligently avoided. The m distinct prefix codes 
(ignoring complements) are called kernels. The property of keeping 
m small is called the sywmetrization property; it is choice-independent. 
Secondly, if all code words v(i | j) for a given t end in the same digit, 
irrespective of /. then first-stage synchronization is greatly enhanced. 
This property of making code-word endings uniform by column (as in 
Table II) is called the columtiization property. (It is more or less 
choice-independent.) A third property involves the number of different 
symbols to which a given code word can correspond, (In the example 
of Table II, half of the code words occur only for one value of i each.) 
Second-stage synchronization is greatly aided by having almost all code 
words occur only for a relatively small number of different symbols, 
avoiding having each occurrence correspond to a different symbol S<i)* 
This is called the association property, and is also choice-inde|iendent. 
It should be noted that none of the above properties is necessary for 
obtaining finite lag codes; however, these properties are found to be 
helpful in achieving low lags. Although the example of Table V is a 
(complemented) one-kernel code, it violates both the columnization 
property and the association property, (Note that each code word 
occurs only twice, but always for different symbols.) Its lag J, though 
finite, is quite horrendous (around 200). with the shortest synchroniz- 
ing sequences being of length ten (e.g.. 0101000100). It thus compares 
badly with the code of Table II with respect to its lag. (It even coin- 
pares badly with the worst coluiniibed one-kernel finite-lag code using 
the given prefix code, for which J = 23.1,) 



Table IV— Example of a Code with J ~ « by Theorem 1(b) 
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Taule V— A Bad Finite Lag Code 
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An example of a two-kernel code which satisfies the columnizatiun 
property and the association property is shown in Table VI. The code 
words which occur for only one value of i are indicated by asterisks. 
This code has J = 8.9, with synchronizing sequences including 0011 
<forB) and 11100 (forC). 

In a columnized code, the set of symbols S(i) for which all code 
words end in "0" ("1") is called the 0-set (1-set). In the example of 
Table VI, the sequence 00 (among others! guarantees the end of a code 
word corresponding to the 0-set A, C or E {i even) , while a 1 1 1 guaran- 
tees the end of a code word corresponding to the I-set B or D (i odd}. 
These are two of the first-stage synchronizing sequences. Having re- 
duced the ambiguity to a 0-set symbol or to a 1-set symbol, the associ- 
ation property within these sets is of great aid to second-stage synchro- 
nization. For example, the code words which optimally satisfy the 
association property (e.g., those with asterisks in Table VI) themselves 
act as second-stage synchronizing sequences in this example. Associa- 
tion is especially helpful to second-stage synchronization if the prefix 
code is the same for each symbol S(j) in the 0-set, and similarly for 
the I-set. This is the bifurcation property of columnized codes, that 
the 0-set and the 1-set use one prefix code each (although the two 
codes may be identical). Note that this property is found in the code 
of Table VI, in which two distinct prefix codes are used. If present, the 
bifijrcation property implies that the symmetrization property is met 
with m = 1 or 2. (By definition, bifurcated codes must be columnized.) 

An example of a one-kernel code satisfying all four of the above 
properties is given in Table VII. Apart from 00 and 01, no code word 



Table VI-A Two-Kernel Example, J = 8.9, /' - 4, J" - 5,1 
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Table VII— A One-Kehnkl Example, J - 20,2, / = 8, J" = 147 
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occurs for more than two symbols .Slil (i.e., in more than two 
columns! . The lag is seen to he J = 20/2. 

5*3 Balanced Codes 

If the columnization property is to be achieved in a complemented 
sequential code, then any kernel prefix code and its complement must 
each have the same number of code words ending in aero (or one). 
Consequently, the prefix code land its complement) must have half 
of its code words ending in each digit. Such a prefix code is called a 
balanced code, and of course must have an even number of code words. 
The codes of Tables II and VII are examples of balanced prefix codes 
used in complemented and uncomplemented one-kernel codes, respec- 
tively. 

Theorem S: For every net of code-word Ungtks with n even for 
which there exists on exhaustive prefix code, there exists at least one 
balanced prefix code. 

Proof: A simple proof involves a construction procedure during 
which the numl>er of code words ending in "0" differs by at most one 
from the number of code words ending in ™1 M . Thus since n is even, 
the resulting code is balanced Such a procedure is easy to construct, 
but is omitted here since it does not contribute to a basic understand- 
ing of the paper. [It can in fact be shown that the ratio of balanced 
exhaustive codes to all exhaustive codes is asymptotic for large n to 

l/M*.] 

A few balanced exhaustive codes are shown in Fig, 11, including one 
for each structure function N(d) with n = 4 and ti. In each of these 
cases, the code shown has the smallest possible lag / among all bal- 
anced codes with the given N(d). 

If the sequential code is not complemented* the kernels need not be 
balanced. However, the use of balanced prefix codes as kernels is 
highly beneficial. It greatly enhances flexibility in the assignment of 
the w{% ;) according to the needs of synchronisation (e.g., via colum- 
nization and association!, coding complexity and compression. 
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Fig. II— Some balanced codes for n 5 fl. 



5.4 Analysis of First-Stage Synchronization 

Another property emerges from considering the relation between 
the lag J of a bifurcated sequential code and the lags of the kernel 
codes. For an uncomplemented one-kernel code, first-stage synchroni- 
zation is guaranteed by the synchronizing sequences of the kernel, i.e., 
with the lag / of the kernel. Thus this lag should clearly be small. On 
the other hand, for a bifurcated complemented one-kernel code, the 
lage / is not relevant to first-stage synchronization. Instead the mu- 
tual lag /' of a prefix code and its complement is needed. The mutual 
tag of a O-set code and a 1-set code is obtained by considering the state 
diagrams of both codes. Imposing the restriction that a code word 
ending in "0" CM") is followed by a O-set (I-set) code word, these 
two state diagrams become one just as the four diagrams in Fig. 7 
become one. The mutual lag /' is then the lag of this combined state 
diagram, obtained from the mutual synchronizing diagram, i*e., the 
synchronizing diagram for the combined state diagram. Terminal nodes 
consist of first-stage synchronisation, i.e., solely of O-set or 1-set code- 
word ends. (Note that the mutual lag is partially choice-independent, 
depending on the choice of the O-set and 1-set prefix codes, but not on 
the actual tf(i ;).) Mutual lags are relevant primarily for balanced 
prefix codes and their complements, as used in (one-kernel) bifurcated 
complemented codes; this case is assumed unless otherwise specified. 
They are also meaningful for two-kernel bifurcated codes, for which 
the mutual synchronization diagram also guarantees first-state syn- 
chronization. (For example, the two prefix codes used in Table VI have 
/' = 4.) Note that the mutual lag of a code with itself is /' = / (since 
the O-set code and the 1-set code are identical). 

Consider as an example the prefix code {0, 11. 100. 101) used in 
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Table II. Suppose that it is used for the 0-set symbols A and C (Fig. 
12a), while its complement is used for the 1-set symbols B and T) (Fig. 
12b). The state 0(1* corresponds to a code word ending in "0" ("1"), 
and implies that the next code word is taken from the 0-set (1-set) 
code. (Note that irrespective of the actual ir(i | ;), the occurrence of a 
00 or a 11 suffices to guarantee the end of a code word, as in Fig. 8.) 
The mutual synchronization diagram is given in Fig. 12c, and has 

/' = 3. 

Values of the mutual lag /' are given in Fig. II for each code shown 
there and its complement. In each case* the code shown has the smallest 
value of /' for any balanced code with the given set of code-word 
lengths N(d). A numerologicat curiosity is provided by Fig. II: for 
each N(d) t the indicated best value of /' is precisely one greater than 
the best value J* of / attainable by any code (unbalanced, in fact) 
with the given A r (rf>. Since this curiosity is true of all A f (d) for ex* 
haustive codes with n ^ 6, including those for odd n, it seems highly 
likely for all n y for all N(d) for which exhaustive codes exist. 

|It should be noted that the value of /' for a given prefix code and 
its complement is obtained with the given code as the 0-set code. If 
it is used instead as the 1-set code, the resulting value of /' is the 
mutual lag of the complemented prefix code, and is designated by I\ 
Values of I s arc also shown in Fig. 11. It is seen that /' and V are not 
usually the same.) 

The kernel lag */' of a bifurcated code is then defined as the synchro- 
nization lag of the first stage. For a one-kernel uncomplemented code, 
J* = /; for a two-kernel (uncomplemented) code, or a one-kernel com- 
plemented code, J f = /'. The kernel lag property then states that the 
kernel lag J' of a bifurcated code should be small, in order to help mini- 
mize the lag J. 
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Fia. 12 — Exumuk of firsWUge synchronization in a complemented code with 
/• = 3. 
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&.& Further Properties of Bifurcated Codes 

The second-stage lag nm y ' >e approximated by the structural lag J 
which assumes first-stage synchronization, and results in a known 
symbol (at the end of a code word!, It is obtained by averaging the 
second-stage lags, A' beginning with the 0-set and K' beginning with 
the 1-sct; A* ami K f are weighted according to the probabilities of 
reaching the 0-set and 1-set, respectively, in the mutual synchroniza- 
tion diagram* (In the example of Tabic VI, for example, it is seen that 
these weights are J and J.) The structural lag property states that this 
lag should ho as small as possible. A rough measure of J" may be 
made independent of the prefix code by assuming all code words of 
equal length, and the U-set and 1-set equiprobablc. 

Theorem 3; For a bifurcated code, 

Proof: The theorem follows from the definitions* Inequality occurs 
when, in the sequential synchronization diagram, first-stage synchro- 
nization occurs in at least one case as the result of an advantageous 
subset of either the 0-set or the 1-set By "advantageous" is meant 
a subset which accelerates second-stage synchronization. The value of J' 
gives precisely the first-stage synchronization lag in any event- When 
there are no such subsets, equality holds, as in Fig, 8- (For example, 
the code of Table VI has J' = 4, J*' - 5.1 and J - 8.9; the sequence 
1100 guarantees not just the 0-set, but specifically a "C" or an "E" 
at its end. The code of Table VII also has such subsets. These subsets 
are obtainable from the firet-stage synchronization diagram, and may 
be used to calculate the exact second-stage lag J~ based on the subsets 
with their appropriate weights, rather than just on the 0-set and the 
1-set Then J - J 9 + J~. In all cases J" £ .T. Also, if J" is infinite, 
then so is */~.) 

Theorem 4-' A bifurcated code has J = *> if and only if either J' 
Or J" is infinite. 

Proof: The "only if" follows as a corollary of Theorem 3. The "if" 
requires two cases. If J" = oo, then J - *> since the 0-set and 1-set 
are subsets of the set of all states, on which J is based. If J' = *>, 
first-stage synchronization is never reached* Since a bifurcated code 
must go through this stage in order to reach second-stage synchro- 
nization, and since J' is the true value of first^stagc lag even when 
there is inequality in Theorem 3, it follows that J — *> - QED. (Note 
that in Theorem 4 J" may be replaced by J .) 
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In many symmetric cases, the 0-set and the I-set are equivalent 
structurally, and K = K' = J". This occurs in the example of Table II t 
since the code obtained under the transformation A « D, B » C is 
the complement of the Table II code (a complementary reflective 
symmetry). It also occurs in Table VII, since the O-set and 1-set sym- 
bols are paired with identical encodings. Such symmetric cases are 
advantageous for encoding/decoding and analytic simplicity* 

In certain cases it is desirable lo use complemented bifurcated codes. 
If the above numcrological curiosity is true in general, there is essen- 
tially no sacrifice in the 1h*sI V of a balanced code compared to the 
best 1 = 1* for the given Xi<f\. That is, first-stage synchronisation is 
just as rapid for these complemented codes, while avoiding the diffi- 
culties arising from the use of an unbalanced code. Further, considering 
only balanced codes for a given Nld), the best value of /' is often 
better than the best value of / (cf- Fig. 111. Thus the overall lag is 
frequently better* Furthermore, the flexibility of compression available 
with the complemented codes is often greater- For example, consider 
again the code of Table II, this time only in terms of its code-word 
lengths. Using an exhaustive prefix code, it is impossible to have length 
one on the i = / diagonal with an uncomplemented one-kernel code 
unless the columnization property is violated; this in turn may greatly 
increase the lag of the code. 

(Surprisingly, prefix codes with / = =o are not altogether useless. If a 
one-kernel code uses a block code or a code the greatest common divi- 
sor (gcd) of whose lengths is greater than one, then it follows that 
J' = J = oo. It is interesting to note, however, that many uniformly 
composed codes and anagrammatic codes (with / = oo) have Y < »» 
and thus may give rise to one-kernel codes with J < oo, assuming 
J* < oo. The smallest possible finite exhaustive anagrammatic code 
with /' = /* = oo has n = 18, and is its own complement. An infinite 
example with the same properties is provided by the self -complement- 
ing anagrammatic code of Fig. 6c. Thus the use of such prefix codes 
in a one-kernel sequential code must result in J = oo, by Theorem 4.) 

VI. GOOD STRUCTURE FOR LARfiE SEQUENTIAL CODES 

The codes of Tahiti II and VII are rather small examples, albeit 
good ones, of one-kernel codes. Since great compressions arc found 
primarily in large codes, the next question is whether low lags can be 
achieved for large codes without sacrificing coding simplicity and 
compression. 
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Assume for now that it is possible to approximate the second- 
order probabilities associated with the code words irfi" | j) as a function 
of k = i — j (mod n). and hence that from a compression point of 
view a difference code is meaningful, (This assumption will be general- 
ized below.) Now consider the general n-level one-kernel encoding 
framework shown in Fig. 13. The difference A" is modified as a function 
of } t and the result k* is encoded by an ordinary prefix encoding 
Wlk*\ . If the code is a complemented code, the result is complemented 
(e.g., if j is odd). (Otherwise the complementing circuitry is not needed, 
as in Table VII. I The corresponding decoder is shown in Fig. 14. 
(Again (he complementing circuitry may not be required.) In many 
cases the demodifieation circuitry of Fig. 14 is identical to the modifi- 
cation circuitry of Fig. 13, with the input and output interchanged. 
Note that the codes of Tables II (see Figs. 9 and 101 and VII are 
examples of codes amendable to this framework. The incremental cost 
of encoding and decoding compared to difference codes is embodied in 
the modification logic and the trivial complementing logic; as long as 
tho modification logic can be kept simple, the incremental cost is low. 
The compression can in general be made at least as goal as the differ- 
ence code. 

If a balanced code is used, the framework defined by Fig. 13 makes 
it easy to satisfy the columnizution property, the symmetrizntion 
property (with m = II, and the bifurcation property. The choice of a 
prefix code is dictated by the kernel lag property, and by the condi- 
tional probability distribution of the symbols S(t), given S(j). The 
suitable structure is dictated by those probabilities, mitigated by the 
structural lag property, and by the complexity of the modification logic 
desired. Considerable experimentation has shown that the properties 
discussed here are central to the construction of good codes. Columni- 
zation and the choice of prefix codes with small J' greatly facilitate 
first-stage synchronization. Association greatly influences second-stage 
synchronization. Bifurcation greatly simplifies coding complexity and 
improves second stage synchronization. The use of balanced codes is 
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Fjr. 13 — Generalized encoder for one-kernel code*. 
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Fip. 14 — Generalised decoder for om-kernel rodrs. 

very helpful If the ihiiiiIht n of messages is odd. however, it may be 
desirable to use n two-kernel code as in Table VI to achieve colurnni- 
zation. 

If the conditional probabilities are strongly asymmetric and/or 
not approximately reprcscntahle hy difference probabilities, it may 
he advantageous from a compression standpoint to use a multi-kernel 
code. Alternatively, or additionally, it may he desirable lo eliminate 
the difference/sum circuitry in Figs, 13 and 14, and to deal directly 
with the i and j. The properties described in this paper are equally 
relevant in such eases. No restrictive assumptions have been made 
regarding the first-order probabilities. It is also unnecessary for O-set 
and 1-sct code words to match even and odd i ( respectively; this is 
merely a descriptive convenience, 

SJ Lartje Balanced Codes 

In order to enhance the kernel lay pro|>erty for codes with large », 
it may be desirable to use truncated systematic prefix codes rather 
than optimal prefix codes as kernels. In this way it is possible for / to 
remain small as n increases. Several examples of such codes which may 
easily be truncated to give balanced codes are summarised in Fig, 15, 
along with their structure functions N{d) and I heir lags A /' and /\ 
The best lag /• for any code with the specified A'lrfl is also given, 
along with the random average code- word length L for Sid). The 
selection of efficient codes for compression purposes is considered in 
Rct\ L 

There exist many classes of codes for which J remains finite (and 
in fact quite small I as n increases without limit. A simple (rather 
extreme) example is indicated in Table VIII for n = 8- The code is 
columnized, complemented, bifurcated, and maximally associated with- 
out being trivial (It assumes very high probability of / = ; for com- 
pression purposes.) To avoid confusion, the symbols are given as ,4 to 
H for i and j from to 7; the integer k is given to indicate the occur- 
rence of the code word IP(A'), If; is odd, the complementary code word 
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Fig* I&— Some systematic prefix codes easily truncated to balanced codes. 

JV'(fc) is used, ami the table is read from (he bottom up. The code 
thus has complementary reflective symmetry. If W{k) = 0, II, 100, 
1011, 10100, lOlOllp 1010100, 1010101 for k = 0, 1, •**, 7, then 
J = 8.62. This code is readily extended to arbitrary even n with a 
similar pattern of fc's: for each even j other than 0, fc = for t = ;", 
fc = ) for i = 0; otherwise fc = i; code words for odd ) are then 
specified by the complementary reflective symmetry. The modification 
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logic of Figs. 13 ami 14 is thus exceedingly simple, and the comple- 
menting logic is again a single EXCLUSIVE OR gate as in Fig, 10- 
Tho limiting prefix code is code Oi) of Fig. 15. Analysis shows that a 
sequential code with ? m P = 3, J" = 4| 1 + % + Ha + % % + ttiiH 
+ ■ - ■] < 5.7, J < 87 exists for every even n. (For suitably skewed 
distributions, the compression factor arising from this code approaches 
the base two logarithm of n.l This is an example of a class of codes for 
which the differential circuitry is not relevant; it is easier to encode 
directly as fc = t, with (ft - 11/2 pairs of exceptions. Variants for 
which the 1-sct is treated in this fashion while the 0-set is treated 
differentially as before are also easily implemented. These variants 
provide a very simple structure which results in low J and simple 

circuitry. 

[A word of caution is in order when considering sequence* of ex- 
haustive prefix codes approaching systematic codes in the limit: the 
limit of a sequence of values of / (or V\ thus obtained is normally 
somewhat larger than the value of / lor U\ for the limiting systematic 
code. (The previous example is an exception with respect to /'.) Code 
(b) of Fig. 15 is an example in point, with / = P = 5. The limit of the 
value of / {or/') for the codes 01, 10, 001, 110,000,111 (/ = /'= 101, 
01, 10, 00I ( 110, 0001, 1110, 0000, 1111, (/ = /' = 81, etc, is seven, 
although this sequence approaches the systematic code (with / = /' 
5) in the limit. In general, truncated systematic codes have better 
synchronization properties than their finite exhaustive approximations, 
since the values of / and /' are essentially unaffected by truncation. 
For large codes, there is little if any noticeable degradation in com- 
pression caused by using truncated infinite codes,) 

The use of definite codes is suggested since their lags are minimal 
(/ = L), while I > L for non-definite codes. However, non-definite 
codes can be quite good. Codes (a I and (cl in Fig- 15, for example, 

Tabu VIII— Example of a Coon Com from an Infinite Class of 
Codes fok Wium J < 8.7 for all n 
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have X(d) for which definite codes exist (whence /* = L). In these 
cases, /' = L + 1, recalling the above numerological curiosity. No 
counterexample to this curiosity is known among N{d) for which 
definite codes exist, or even among all systematic codes. In general, 
there are many non-definite cases for which / or /' is quite close to 
L + 1. Thus in these cases the non-definite systematic codes do almost 
as well as comparable definite codes in achieving first-stage synchroni- 
zation. Subsequently the non-definite codes may he much better in 
reaching the second stage. Definite codes are necessarily completely 
unbalanced, 1 ** with all code words ending in the same digit; therefore 
a complemented columnized code is impossible. Although a non-com- 
plemented code is thus automatically columnized if it uses a definite 
prefix code as its kernel, second*stage synchronization often must 
begin with the set of symbols, not just half of them as in the case of a 
balanced code. However, some definite codes are balanced on two 
distinct terminal sequences, e.g., on the next to last digit of each code 
word or on some earlier digit position. In these cases, it is possible to 
columnize on the two distinct terminal sequences. As an example, con- 
sider code (d) of Fig. 15. This is a definite code defined by the set of 
sequences (0100,01101, with N(d) =0022448 12 24 •■■ . For each 
length there are exactly as many code words ending in 00 as in 10- If 
the sequential code is columnized according to the last two digits of 
each code word into a 00-set and a 10-set, first-stage synchronization 
results in one of these two sets, as in the balanced code situation. Since 
/ = L = 8, this code has synchronization properties excelling any non- 
definite code with the given A*(d). 

6.2 Guidelines for Choosing Good Encodings 

Because of the perversity of the three-dimensional tradeoffs among 
compression, complexity and synch ronizability, it is pointless to try to 
give a specific algorithm for choosing the best encoding for a given 
application. Optimality in any one dimension is of little concern, for 
slight sacrifices in any of these dimensions often result in great savings 
in the other two. Besides, no sensible cost metric is known. Nevertheless, 
the techniques of this paper provide a set of guidelines for the con- 
struction of good codes and good encodings. 

The first step in selecting a sequential encoding is to establish for 
each j the code-word lengths which are optimal for the code word 
w(i | ;) in the prefix code for the given j, based on the conditional prob- 
abilities of S(i), given S(j). This may be done simply using any variant 
of the Huffman algorithm 11 which derives the code-word lengths. In- 
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spection of the matrix of lengths thus obtained (i.e., of the set of n 
structure functions, one for each » indicates what kind of symmetries 
the code might reasonably have, e.g., whether a reflective symmetry is 
in order, and whether the code can be a one-kernel code- A comple- 
mented bifurcated code may be required for compression reasons, as 
in Table II. The next step is to choose the basis codc(s), considering the 
set of lengths and the first-stage lag ■/'. For large codes, techniques of 
Ref. 1 may be required to aid code selection- The encodings should then 
be arranged to have the columnization and association properties to 
help minimize the first- and second-stage lags, respectively. The de- 
sired code-word lengths should be taken as suggestive rather than as 
mandatory; slight departures from these lengths are generally not 
harmful to compression (especially among large lengths), and may 
help greatly in decreasing J~ t Care should be taken to avoid having 
short synchronizing sequences occur only as the result of unlikely 
sequences of source messages. 

VII. CONCLUSIONS 

The object of this pajier is to present codes with low redundancy, 
reasonable complexity and intrinsic error tolerance, within a single 
class of codes designed for that purpose. The approach taken is by 
no means the only one, although the codes exhibited here seem quite 
powerful in view of their capabilities. In combination with some 
additional redundancy (e.g., as in Refs. I, 4, 17 and 18* to he used 
for error-detection and/or correction and/or forced framing, the in- 
trinsic properties describe*! hero may be used to great advantage. 

The sequential prefix codes of this paper can be useful for a wide 
range of conditional probability distributions. Quantized video picture 
information provides an example of a class of distributions 1 *" 31 to which 
these codes seem suited, with respect to compression and synchroniza- 
bility, as well as to the meaningfulncss of tolerating errors for a short 
period of time. Such distributions are strongly geometric,-" for which 
the codes presented here are naturally applicable. 

The techniques described here art 1 also applicable to other com- 
pression situations. Examples include run-length coding (which has the 
same essential synchronization problem as differential coding* and 
predictive, averaging or smoothing schemes, or combinations thereof. 
In addition, the same techniques are applicable when code words 
w{i | ;) need not be provided for many of the transitions, for example, 
when only relatively small differences are possible. 
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In conclusion, a wide variety of sequential prefix codes has been 
presented, with a considerable range of tradeoffs among coding com- 
plexity, compression and synchronizability. By not insisting on opti- 
mally in any of these, it is possible to obtain codes which are highly 
satisfactory in all of these respects at the same time* 
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